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Mathematics, Engineering & Science Achievement (MESA) is a national, pre-college academic pro- 
gram that helps underrepresented students develop science, technology, engineering, and math 
(STEM) skills. It also helps them learn invention and 21st-century skills, such as marketing, commu- 
nication, collaboration, and critical thinking. The Oregon chapter of MESA was established in 1985 
as an after-school program. It seeks to connect STEM and 21st-century skills to students’ real lives 
and aspirations, with the goal of being better prepared to graduate from high school, enroll in 


postsecondary education, and find meaningful work in the global marketplace. 


At the request of Oregon MESA and with the support of the Meyer Memorial Trust and the Lemelson 
Foundation, researchers at Education Northwest examined the program’s impact on the educa- 
tional outcomes of participants in Oregon middle schools, evaluating whether MESA met its goal 
of increasing high school graduation. To do this, we collected student-level data from the Oregon 
Department of Education (ODE) and used statistical methods to identify a group of students with 
similar characteristics to MESA students but who did not participate in the program. We compared 


these students’ test scores and graduation rates to those of MESA students. 


We found that MESA students, on average, had higher science 
test scores and higher graduation rates than their peers. Howev- 
er, after accounting for the differences between MESA students 
and their peers (such as their previous academic achievement 
and personal characteristics), we found that only one measure, 
the likelihood of graduating from high school, was statistically 
significant. This demonstrates that MESA has a positive effect on 
high school graduation—a central goal of the program. 


Using a quasi-experimental study 
design, we found that MESA students 
were significantly more likely to 
graduate from high school than 
their peers who did not participate 
in the program. This relationship 
suggests MESA participation has a 
tangible and important effect on high 
school graduation. 
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Our analysis accounts for many of the factors that influence students’ academic outcomes, such 


as their prior academic performance, demographic characteristics, and socioeconomic status. 


However, we could not account for other important factors, such as students’ motivation, parents’ 


engagement, and quality of instruction. We encourage readers to think critically and keep these 


factors, as well as their own experience, in mind as they interpret the findings. 


HOW DID WE STUDY MESA? 


We compared MESA students to a matched sample of Oregon students who did not participate in 


the program but were otherwise similar in all observable ways. Specifically, we compared their per- 


formance on the statewide grade 8 and high school (typically grade 11) English language arts (ELA), 


math, and science assessments, as well as high school graduation rates. 


What is a matched sample? 


A matched sample is a way of creating a comparison group that can show what might have hap- 


pened if MESA participants never joined the program. Comparing academic outcomes between 


MESA participants and their matched peers demonstrates the impact of the program. Matching en- 


sures we are comparing students who had similar characteristics and academic achievement before 


they participated in MESA. 


We used coarsened exact matching to identify a comparison 
group of students who did not participate in MESA but are 
otherwise similar to MESA students in all observable ways. 
This is called a quasi-experimental design because instead of an 
experimental design—which would randomly select which 
students participate in MESA and which students do not—we 


used statistical methods to identify a comparison group. 


However, unlike random assignment, a matched sample 
cannot guarantee that participants were not in some way 
different from non-participants. Although we can test to see 
whether the matched sample of students and MESA students 
were similar before the latter joined the program (they were), 
we cannot do this with characteristics, such as students’ moti- 
vation or parents’ engagement, that are not measurable from 


administrative datasets. 


Why did we use matching? 

Random assignment is the most rigorous 
method of determining the impact of a 
program. Random assignment of enough 
students ensures all characteristics that could 
affect student performance are balanced 
and equal between students who participate 
in the program and those who do not. 


However, random assignment is not always 
practical or ethical. Since we could not 
randomly assign students to participate in 
MESA, we used statistical methods to create 
a matched sample of students who are simi- 
lar to program participants in all observable 
ways—except that they did not participate 
in MESA. 


This underscores the limitation of using quasi-experimental design and matched samples—we can- 


not completely attribute differences in student outcomes to the program itself. ln other words, we 


are certain that the students who participated in MESA were more likely to graduate from high 


school than their peers, but we cannot be certain that participation in MESA was the only cause 


of the difference; there may have been other causes that we did not account for. 
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Figure 1. Description of study cohorts 
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Note: This figure shows the distribution of grades and outcomes over time for the three cohorts of students who were 

included in the study. Each cohort was defined by the year they first attended grade 6, which was the first year of possible 
MESA participation in this study. We identified a comparison group based on grade 5 test scores, program participation, 
and demographic characteristics. We then compared outcomes in grade 8, in high school, and after graduation. 


Who were the MESA students in the study? 


To determine which Oregon students were MESA participants, we matched program records 
to ODE administrative files using students’ names, birthdates, and school districts. We identified 
431 students who participated in MESA in grades 6-8 from 2009-10 to 2013-14 and who had the 
minimum necessary data to be included in the study, such as a state test score in grade 8 or high 


school (see the appendix for the matching procedure). 


MESA students in the study sample attended four Oregon school districts: Portland Public Schools, 
Parkrose School District, Hillsboro School District, and Beaverton School District. The students were 
members of three cohorts, as defined by their first year of grade 6, which was their first year of pos- 
sible MESA participation. Depending on their cohort, these students took grade 8 and high school 
state tests, potentially graduated from high school, and matriculated to a postsecondary education 


institution in different years (figure 1). 


Who were the comparison students? 


Students in the comparison group did not participate in MESA but were otherwise similar in observ- 
able ways to program participants. Specifically, they were in the same grades and school districts, 


and they had similar academic achievement and demographic characteristics. 


We matched MESA students to comparison group students in districts that offered MESA. We used 
nine covariates (student-level data points that possibly predict student outcomes) to ensure the 
students were well matched: race/ethnicity; gender; grade level; state assessment results in ELA, 


math, and science in grade 5 (one year before students had the option of participating in MESA); and 
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Postsecondary 


eligibility for three federal programs: English learner services, special education services, and the 


free or reduced-price lunch program (a proxy for poverty). 


With these characteristics, we matched the 431 MESA students 
to one or more individuals who were statistically similar to 
them from a comparison group of 11,839 non-MESA students. 
We combined all MESA students into one group, regardless of 
when and for how many years they participated in MESA. 


In grade 5—before they participated in the program—MESA 
students typically had higher test scores than the statewide 
average in Oregon (table A1 in the appendix). MESA students 
were also more racially and ethnically diverse than other Or- 
egon students, and a higher percentage of MESA students 
were English learner students or eligible for the federal lunch 
program (table A2 in the appendix). 


Why did we use standard scores? 
Since we are interested in how MESA stu- 
dents perform in comparison to their peers 
over different years, we cannot directly com- 
pare scaled scores. Instead we use standard 
scores, which illustrate the distance between 
groups of students on a normal distribution. 
One standard score is the same as a standard 
deviation and translates to about 34 percen- 
tile points above or below the mean. 


MESA students also had slightly higher assessment scores than the comparison group in grade 5 (ta- 


ble 1). However, the differences were not statistically significant (table A3 in the appendix) and were 


close enough to establish baseline equivalence, which means that MESA students and the comparison 


group students had characteristics that were comparable (equivalent) before the former participated 


in MESA (at baseline). 


Table 1. Baseline results from grade 5 were well matched 


MESA students Comparison group students 
(n = 431) (n = 11,839) 

State test standard scores (OAKS) 

ELA 0.11 0.10 
Math 0.26 0.23 
Science 0.12 0.09 
Program participation (ever) 

English learner 35% 29% 
Federal lunch program 76% 65% 
Special education 23% 15% 
Demographic characteristics 

Female 49% 49% 
Hispanic/Latino 32% 31% 
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MESA students Comparison group students 


(n = 431) (n= 11,839) 
Asian & Pacific Islander 15% 8% 
African American/Black 10% 5% 
White 35% 50% 
Other* 8% 5% 


* Other includes students who identify as multiracial, Alaska Native, and American Indian. 


Note: N-values vary by test and group; exact numbers are in table A3 in the appendix. Results do not account for weights 
assigned during the matching process and do not control for prior achievement, program participation, or student demo- 
graphic characteristics. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 


How did we compare MESA students to their peers? 


We used regression analysis to examine the relationship of MESA participation to student perfor- 
mance. Regression allowed us to estimate the contribution of MESA on student learning based 
on students’ performance on state standardized achievement tests or its impact on graduation. 
Regression also allowed us to compare student test scores or outcomes while accounting for other 
factors (such as prior achievement and demographic characteristics, as well as eligibility for special 


education services, English learner services, and the federal lunch program). 


WHAT DID WE FIND? 


MESA students were significantly more likely to graduate from high school than their peers who did 
not participate in the program. We did not find any other statistically significant differences between 
MESA students and their matched peers on the statewide grade 8 and high school ELA, math, and 
science assessments. 


MESA students had higher science assessment scores in grade 8 and high 
school than their peers, but these differences were not statistically significant 


On average, MESA students had higher science test scores than their matched peers in the com- 
parison group (table A3 in the appendix) in both grade 8 (figure 2) and high school (figure 3). MESA 

students’ ELA test scores were higher than the comparison group in grade 8, and the difference was 
nearly indistinguishable in high school. MESA students’ math scores were lower than the compari- 


son group in grade 8, and the difference was indistinguishable in high school. 
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Figure 2. On average, MESA students had higher ELA and science test scores in grade 8 than their 


peers, but these differences were not statistically significant 


ELA standard score Math standard score Science standard score 
(grade 8) (grade 8) (grade 8) 


@ MESA students © Comparison group 


Note: These figures show average standard scores on Oregon state assessments. The assessment scores apply weights 
assigned during the matching process but do not control for prior achievement, program participation, or student demo- 
graphic characteristics. Table A3 in the appendix provides full results. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 


Figure 3. On average, MESA students had higher science test scores in high school than their peers, 
but these differences were not statistically significant 


ELA standard score Math standard score Science standard score 
(high school) (high school) (high school) 


@ MESA students @ Comparison group 


Note: These figures show average standard scores on Oregon state assessments. The assessment scores apply weights 
assigned during the matching process but do not control for prior achievement, program participation, or student 
demographic characteristics. Table A3 in the appendix provides full results. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 
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However, after accounting for prior achievement, program enrollment, and demographic character- 
istics, these differences were not statistically significant (table A5 in the appendix). This means the 
differences between MESA students and the comparison group could have been due to chance or 


normal population variation rather than program participation. 


MESA students had significantly higher graduation rates than their peers 


MESA students were more likely to graduate from high school than comparison group students. 
Specifically, 87 percent of MESA students graduated from high school in four years compared with 


73 percent of the comparison group (figure 4). 


Figure 4. MESA students had a higher four-year graduation rate than their matched peers 


MESA students 


Comparison group 


Note: The graduation outcomes apply weights assigned during the 

matching process but do not control for prior achievement, program 

participation, or student demographic characteristics. Table A4 in the 
appendix provides full results. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) 
and ODE data (2008-09 to 2017-18). 


We found that these differences were statistically signifi- 
cant (p < .001), even after accounting for prior achievement, 
program enrollment, and demographic characteristics (ta- 
ble A6 in the appendix). The difference in the four-year 
graduation rate between MESA students and comparison 
group students was also large. Specifically, we found that 
students who participated in MESA had 3.13 times the 
odds to graduate from high school in four years than the 
odds of students who did not participate in MESA (table 
A6 in the appendix). 


Of the 335 MESA students who graduated from high school, 
all but one did so in four years (table A4 in the appendix). In 
contrast, among the 8,457 comparison group students who 
graduated from high school, 756 (9 percent) needed more 


than four years. Even with this additional time to graduate, 
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Whatis an odds ratio? 

There are different ways of expressing the 
outcomes of a regression analysis, but none 
of them are easy to understand. Findings from 
a logistic regression (which examine binary 
outcomes, such as whether or not a student 
graduates) compare the likelihood of an event 
occurring among different groups. One way of 
describing likelihood is an odds ratio. 


An odds ratio describes the probability that 
an event will occur for one group relative 
to another group. In our analysis, an odds 
ratio of 1 would mean that two groups of 
students have the same odds of graduating 
from high school. An odds ratio of more than 
1 shows higher likelihood of graduating from 
high school, and an odds ratio of less than 1 
shows a lower likelihood. 


We found an odds ratio of 3.13, which means 
a MESA student is more than three times as 
likely to graduate from high school in four 
years compared with their peers who do not 
participate in MESA. 


MESA students were still significantly more likely to ever graduate from high school than comparison 


group students (table A6 in the appendix). 


Finally, we did not find any interaction effects, that is, statistically significant differences in how MESA 
interacts with graduation outcomes by students’ gender, race/ethnicity, eligibility for the federal 
lunch program, or English learner status. This means participating in MESA is likely to boost the like- 
lihood of graduating from high school in four years for all students. 


HOW DO WE UNDERSTAND THE SIZE OF MESA’S EFFECT 
ON GRADUATION? 


MESA students were 3.13 times more likely to graduate from high school in four years than compar- 
ison group students. Stated another way, we found that participating in MESA appears to increase 
the probability of graduating from high school in four years by 21 percentage points (table A7 in 
the appendix). 


In comparison, being affected by poverty (defined as being eligible for the federal lunch program) 
had a similar magnitude of effect on the likelihood of graduation. However, unlike MESA participa- 
tion, this effect was negative—MESA participants were 21 percentage points more likely to graduate 
from high school in four years than comparison group students, and students affected by poverty 
were 20 percentage points /ess likely to graduate from high school in four years than their peers 


(figure 5, table A7 in the appendix). 


Figure 5. Participating in MESA had the same magnitude of effect on high school graduation 
as being affected by poverty 


MESA participants were more Students who were ever eligible for the 
likely to graduate from high federal lunch program were less likely to 
school in four years than similar graduate from high school than similar 
non-MESA participants students who were never eligible 


Eligible Not eligible 


Comparison 
Group 


Note: See table A7 in the appendix for full results. 
Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 


Education Northwest | Oregon MESA: Increasing the odds of high school graduation 


How do these findings relate to other programs that affect graduation? 


Another way to understand the magnitude of MESA’s effect on four-year high school graduation is 
by comparing it to other programs that are linked to improved graduation rates. We identified one 
intervention in the U.S. Department of Education’s What Works Clearinghouse and one in a recently 

published review of after-school programs (Neild et al., 2019a, 2019b) that appear to have a statisti- 
cally significant impact on high school graduation. MESA’s impact on graduation, measured by an 


odds ratio (3.13), was larger than either of the comparison programs. 


Citizen Schools is an after-school program for low-income youth in grade 8 that includes career 
exposure, high school and college preparation, and academic enrichment. A quasi-experimental 
design study found that Citizen Schools participants were 1.70 times more likely to graduate from 
high school in four years than comparison group students (Acaira et al., 2010). 


Early College is a dual enrollment program that supports underserved students as they take college 
courses while they are still in high school. An experimental study found that students participating 
in Early College were 1.48 times more likely to graduate from high school than students who were 


not selected to participate in the program (Berger et al., 2013). 


Figure 6. MESA’s impact on high school graduation, as measured by odds ratio, was larger than two 
comparison programs 


MESA 


Citizen Schools 


Early College 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18); Acaira et al., 2010; 
Berger et al., 2013. 


How do these findings relate to previous research on MESA? 


In 2016, MESA and Education Northwest partnered to conduct an evaluation of the impact of MESA’s 
after-school program on the educational outcomes of middle school students in Salem-Keizer 
Public Schools (Greenberg Motamedi & Singh, 2016). We found that Salem-Keizer MESA students 
had higher grades in science and math than their peers in 2015-16, as well as a higher-grade point 
average (GPA) in their core ELA, math, science, and social studies classes. Similar to this study, we did 
not find any statistically significant differences in test scores between Salem-Keizer MESA students 


and comparison group students. 
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Grades are generally a better predictor of a student's likelihood of graduating from high school 
(Easton et al., 2017) and matriculating to college (Hodara & Lewis, 2017) than test scores. In retro- 
spect, the higher grades our earlier study found among Salem-Keizer MESA students may have been 
the first sign that they were more likely to graduate from high school than their peers. 


MOVING FORWARD TO POSTSECONDARY OUTCOMES 


MESA and Education Northwest are continuing to work together to understand the relationship 
between MESA participation and attending a postsecondary institution, such as a two-year commu- 
nity college or four-year university. To do this, we will identify MESA students and comparison group 
students who are registered and attending classes at a postsecondary institution in Oregon using 


National Student Clearinghouse data. 


Education Northwest | Oregon MESA: Increasing the odds of high school graduation 


APPENDIX 


This appendix includes details about our matching procedures, analysis, as well as data and 


regression tables. 


Identifying MESA students 


MESA records and ODE data do not share a common student identifier. We used fuzzy matching 

procedures to link MESA students with their ODE records. We used MESA students’ first and last 
name, school district, and date of birth to link the records. Computer algorithms used all avail- 
able information to calculate a similarity score that quantified how alike two records were in the 


different datasets. 


We included MESA records for 497 individual students in grades 6-8 who applied to the program 
and attended two or more MESA meetings from 2009-10 to 2013-14. Of these, we found exact 
matches in the ODE data for 346 records and probable matches for an additional 117 records. There 
were only 34 records for whom no probable matches were found in the ODE data. We also identified 
and resolved duplicate records. The overall match rate was 93 percent, or 463 ODE students from 
497 MESA records. 


Compared with the statewide average, MESA students had higher test scores in all grades and all 
subjects (table A1). MESA students were also more racially and ethnically diverse than other Oregon 
students (table A2). 


Table A1. Unweighted assessment scores for MESA students and all Oregon students 


ELA Math Science 
standard = standard - standard 
score score score 

Grade 5 assessment scores 

MESA students 455 0.11 459 0.26 420 0.11 
Oregon average 171,419 0.01 171,707 0.04 133,391 0.01 
Grade 8 assessment scores 

MESA students 449 0.14 447 0.22 392 0.09 
Oregon average 117,040 0.05 116,915 0.14 104,886 0.00 
High school assessment scores 

MESA students 364 0.24 366 0.31 277 0.15 
Oregon average 103,961 0.23 103,028 0.26 86,709 0.09 


Note: The assessment scores do not account for weights assigned during the matching process and do not control for prior 
achievement, program participation, or student demographic characteristics. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 


Education Northwest | Oregon MESA: Increasing the odds of high school graduation 


Table A2. Unweighted demographic characteristics for MESA students and all Oregon students 


MESA students Statewide average 
(n = 463) (N = 274,936) 

Program participation (ever) 

English learner 35% 22% 
Federal lunch program 76% 63% 
Special education 23% 21% 
Demographic characteristics 

Female 49% 49% 
Hispanic/Latino 32% 27% 
Asian & Pacific Islander 15% 8% 
Black/African American 10% 6% 
White 35% 53% 
Other* 8% 6% 


* Other includes students who identify as multiracial, Alaska Native, and American Indian. 


Note: These demographic characteristics do not account for weights assigned during the matching process 
and do not control for prior achievement, program participation, or student demographic characteristics. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 


Matching procedures 


We were able to use coarsened exact matching procedures to match 431 MESA students (87 percent 
of the original sample of MESA records) to a similar group of comparison students. Students were 


matched in districts that offered MESA using the following student-level variables: 
« Prior achievement on the grade 5 ELA, math, and science assessments 


* Socioeconomic status using students’ eligibility at any time before 2018-19 in Oregon K-12 


public schools for free or reduced-price lunch 


* Special education services eligibility at any time before 2018-19 in Oregon K-12 
public schools 


- English learner services eligibility at any time before 2018-19 in Oregon K-12 public schools 


* Gender identity, either male or female, while enrolled in middle school (other gender 


options were not available in the data) 


« Race/ethnicity using ODE's six categories 


Education Northwest | Oregon MESA: Increasing the odds of high school graduation 12 


Based on these variables, we identified 11,839 students in the comparison group, out of a total of 
187,172 non-MESA students in Oregon K-12 public schools (6 percent). We established baseline 

equivalence of MESA students and the comparison group using Hedge’s g, a statistical test measur- 

ing how much one group differs from another. We found that the differences ranged from 0.09 to 
0.13 standard deviations (table A3). This range of difference meets the What Works Clearinghouse 

standards with reservations for baseline equivalence with statistical controls for the characteristics 
in the analysis (What Works Clearinghouse, 2020). 


We then compared the two groups’ assessment outcomes in grade 8 and high school (table A3) and 


high school graduation rates (table A4). 


Table A3. Weighted assessment scores for MESA and comparison group students 


ELA Math Science 
a standard - standard 2 standard 

score score score 
Grade 5 assessment scores 
MESA students 429 0.11 430 0.26 401 0.12 
Comparison 11,797 0.10 11,812 0.23 11,507 0.09 
Hedge's g 0.13 0.09 0.09 
Grade 8 assessment scores 
MESA students 418 0.14 416 0.22 364 0.09 
Comparison 10,425 0.10 10,395 0.27 9,136 0.01 
High school assessment scores 
MESA students 340 0.23 341 0.30 261 0.15 
Comparison 8,148 0.24 8,004 0.30 6,019 0.07 


Note: The assessment scores apply weights assigned during the matching process but do not control for prior achievement, 


program participation, or student demographic characteristics. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 


Table A4. Weighted high school graduation rates for MESA and comparison group students 


Four-year Graduation Ever Graduated 
Total 
n Percent n Percent 
MESA students 386 334 87% 335 87% 
Comparison 10,572 7,701 73% 8,457 80% 


Note: The graduation outcomes apply weights assigned during the matching process but do not control for prior 
achievement, program participation, or student demographic characteristics. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 
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Testing the association between MESA participation and student outcomes 


To answer the question of whether MESA participation is associated with a change in students’ 


assessment scores in ELA, math, and science (table A5) or graduation (table A6), we fit a series of 


regression models. They included an indicator for MESA participation and controls for achievement 


before participating in MESA. We also included controls for students being eligible for the federal 


lunch program, special education services, or English learner services at any time in any Oregon 


K-12 public school. Finally, we added controls for student demographic characteristics. We found 


that MESA participation was not statistically significant (p > .05) for any of the assessment outcomes, 


which suggests that MESA did not meaningfully improve students’ assessment outcomes. 


Table A5. Regression results for assessment scores 


Grade8 Highschool Grade8 Highschool Grade8 _ Highschool 
ELA ELA math math science science 
MESA participation 0.05 0.00 0.05 0.03 0.10 0.07 
(0.06) (0.06) (0.07) (0.09) (0.07) (0.08) 
Grade 5 achievement 
OAKS ELA 0:67*** 0.59*** 
(0.01) (0.02) 
OAKS math 0.71*** 0.68*** 
(0.02) (0.02) 
OAKS science 0.65*** 0.59*** 
(0.02) (0.03) 
Program participation (ever) 
Federal lunch -0.15*** -0.17*** -0.20*** -0.20*** -0.19*** -0.18*** 
(0.03) (0.03) (0.04) (0.07) (0.03) (0.05) 
Special education -0.20*** -0.27*** -0.15*** -0.14** -0.24*** -0.16* 
(0.03) (0.04) (0.04) (0.05) (0.03) (0.07) 
English learner -0.01 0.08* -0.04 -0.02 -0.07 0.03 
(0.04) (0.04) (0.04) (0.04) (0.04) (0.04) 
Demographic characteristics 
Female Q.17*** 0.09*** 0.09*** 0.04 -0.06* 0.01 
(0.03) (0.03) (0.02) (0.03) (0.03) (0.03) 
Hispanic/Latino -0.14*** -0.11*** -0.14*** -0.12** -0.12*** -0.16** 
(0.04) (0.03) (0.04) (0.04) (0.03) (0.06) 
American Indian & -0.12 -0.38** -0.19 -0.12 -0.19 -0.22 
Alaska Native (0.11) (0.12) (0.11) (0.09) (0.12) (0.12) 
Asian & Pacific 0.05 0.09 0.19** 0.26*** 0.10 0.10 
Islander (0.05) (0.05) (0.06) (0.08) (0.05) (0.07) 


Education Northwest | Oregon MESA: Increasing the odds of high school graduation 


Grade8 Highschool Grade8 Highschool Grade8 Highschool 
ELA ELA math math science science 
Black/African -0.16*** -0.19** -0.17** -0.18* -0.27*** -0.40*** 
American (0.04) (0.07) (0.07) (0.08) (0.04) (0.09) 
Multiracial -0.07 0.00 -0.04 0.05 0.02 -0.02 
(0.06) (0.06) (0.06) (0.07) (0.03) (0.07) 
Variance components 
Constant 017" 031°" 0.23*** 0,32*** 0.26*** 0.20** 
(0.04) (0.04) (0.05) (0.05) (0.04) (0.06) 
Level 2, constant “1.85*** -1.91*** =l6e"* -1.772*** “1.51°** =1,02*** 
(0.11) (0.10) (0.09) (0.109) (0.08) (0.11) 
Level 2, residual -0.47*** -0.42*** -0.42*** -0.335*** -0.46*** -0.27*** 
(0.02) (0.01) (0.02) (0.0182) (0.01) (0.03) 
Observations 10,827 8,474 10,808 8,339 9,255 6,140 


*p < 0.05, **p < 0.01, ***p < 0.001 


Note: Standard errors are shown in parentheses. 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 


MESA participation was statistically significant (p < .001) for both four-year high school graduation 


and ever-high school graduation outcomes, suggesting that MESA participation was linked to 


improved likelihood of graduating from high school (table A6). 


Table A6. Regression results for high school graduation 


Four-year high school Ever graduated from 
graduation high school 
MESA participation 
Odds ratio 3:13*** 1.76*** 
(0.39) (0.25) 
Grade 5 achievement 
OAKS ELA 1.16 1.16 
(0.09) (0.08) 
OAKS math 1.34 1.39*%* 
(0.12) (0.16) 
OAKS science 1.04 1.07 
(0.07) (0.08) 
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Four-year high school Ever graduated from 
graduation high school 

Federal lunch 0.34 1.54*** 

(0.08) (0.28) 
Special education 0.85 0.85 

(0.12) (0.13) 
English learner 1.68 1.54* 

(0.30) (0.28) 


Female 1.24 1.32 
(0.12) (0.13) 
Hispanic/Latino 1.15 0.98 
(0.19) (0.16) 
American Indian & Alaska Native 0.97 0.72 
(0.37) (0.27) 
Asian & Pacific Islander 1.89 1.56 
(0.50) (0.43) 
Black/African American 1.50 1.26 
(0.28) (0.20) 
Multiracial 1.73 1.53 
(0.32) (0.30) 


Constant 3.60 7.64*** 
(0.78) (1.91) 

Level 2, constant 0.23 0.18** 
(0.05) (3.14) 

Observations 10,689 10,689 


*p < 0.05, **p < 0.01, ***p < 0.001 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008- 


09 to 2017-18). 
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We also calculated the marginal effects of MESA on four-year high school graduation. Marginal 
effects are a way to describe the average effect of an intervention on the probability of an outcome 
occurring. In this case, marginal effects tell us how much MESA participation is associated with 


changes in the likelihood that a student will graduate from high school in four years. 


We found that participating in MESA increases the probability of graduating from high school in 

four years by 21 percentage points, controlling for previous achievement, program participation, 
and student demographic characteristics (table A7). Similarly, ever having qualified for the federal 

lunch program—a measure of poverty—decreases the probability of graduating from high school 
in four years by about 20 percentage points, and being female increases the probability of four- 


year high school graduation by about 4 percentage points. 


Table A7. Marginal effects analysis results for four-year high school graduation 


Marginal Standard z-value 95% confidence 
effects error interval 
MESA participation*** 21% 0.02 8.53 0.16 0.26 
Grade 5 achievement 
OAKS ELA 3% 0.02 1.76 0.00 0.06 
OAKS math** 5% 0.02 3.30 0.02 0.09 
OAKS science 1% 0.01 0.63 -0.02 0.03 


Program participation (ever) 


Federal lunch*** -20% 0.04 -4.73 -0.28 -0.12 
Special education -3% 0.03 -1.17 -0.08 0.02 
English learner** 9% 0.03 2.95 0.03 0.16 


Demographic characteristics 


Female* 4% 0.02 2.20 0.00 0.08 
Hispanic/Latino 3% 0.03 0.85 -0.03 0.08 
American Indian & 0% 0.07 -0.07 -0.14 0.13 
Alaska Native 

Asian & Pacific 12% 0.05 2.48 0.02 0.21 
Islander* 

Black/African 7% 0.03 2.18 0.01 0.14 
American* 

Multiracial** 10% 0.03 2.99 0.03 0.17 


*p < 0.05, **p < 0.01, ***p < 0.001 


Source: Education Northwest analysis of MESA (2009-10 to 2013-14) and ODE data (2008-09 to 2017-18). 
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ABOUT EDUCATION NORTHWEST 


Founded as a nonprofit corporation in 1966, Education Northwest builds capacity in schools, families, and com- 
munities through applied research and development. At the request of Oregon MESA and with the support of the 
Meyer Memorial Trust and the Lemelson Foundation, researchers at Education Northwest examined the program’s 
impact on the educational outcomes of participants, evaluating whether MESA met its goals of increasing high 


school graduation. 


Abstract 


Education Northwest partnered with Oregon Mathematics, Engineering & Science Achievement (MESA) to mea- 
sure the effect of program participation by comparing academic outcomes of two groups of students: those who 
participated in MESA and a matched group of peers who did not. All things that we could measure and are linked 
to student academic outcomes—prior achievement, federal program eligibility, and demographic characteristics— 
were the same or similar among individual MESA students and their matched peers. The only observable difference 
among data collected by the Oregon Department of Education is that one group participated in MESA, and the 
other did not. 


We found that MESA students were significantly more likely to graduate from high school than their peers. Spe- 
cifically, students who participated in MESA were 3.13 times more likely to graduate from high school in four years 
than students who did not participate in the program. Stated another way, participating in MESA appears to 

increase the probability of graduating from high school in four years by 21 percentage points. We did not find any 
differences in how MESA interacts with graduation outcomes by students’ gender, race/ethnicity, eligibility for the 
free or reduced-price lunch program, or English learner status. This means MESA is likely to boost the likelihood of 


high school graduation in four years for all students. 
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